Hemlata Channe¶
Project 4: Unsupervised Learning - Clustering¶
Data Description:¶
The data contains features extracted from the silhouette of vehicles in different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.
Domain: Object recognition¶
Context:¶
The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
Attribute Information:¶
● All the features are geometric features extracted from the silhouette. ● All are numeric in nature.
Learning Outcomes:¶
● Exploratory Data Analysis ● Reduce number dimensions in the dataset with minimal information loss ● Train a model using Principle Components
Objective:¶
Apply dimensionality reduction technique – PCA and train a model using principle components instead of training the model using just the raw data.
In [1]:
import numpy as np
from sklearn.linear_model import LinearRegression
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
from scipy.stats import zscore
from sklearn.decomposition import PCA
from sklearn import svm
Task 1: Data pre-processing – Perform all the necessary preprocessing on the data ready to be fed to an Unsupervised algorithm (10 marks)¶
In [2]:
Data = pd.read_csv("vehicle-1.csv")
Data.head()
Out[2]:
In [3]:
Data.describe().transpose
Out[3]:
In [4]:
Data.columns
Out[4]:
In [5]:
Data.info()
In [6]:
Data.shape
Out[6]:
Findings from column description¶
- There are 19 columns with 846 rows in the dataset.
- All the columns are numeric
- The dataset have few null values in many columns 'circularity', 'distance_circularity', 'radius_ratio', 'pr.axis_aspect_ratio','scatter_ratio','elongatedness','pr.axis_rectangularity','scaled_variance',scaled_variance.1','scaled_radius_of_gyration','scaled_radius_of_gyration.1','skewness_about','skewness_about.1','skewness_about.2'
In [7]:
Data.median()
Out[7]:
In [8]:
# replace the missing values with median value.
# Note, we do not need to specify the column names below
# every column's missing value is replaced with that column's median respectively (axis =0 means columnwise)
#Data = Data.fillna(Data.median())
medianFiller = lambda x: x.fillna(x.median() if x.name!='class' else x)
Data = Data.apply(medianFiller,axis=0)
In [9]:
Data.info()
In [10]:
X = Data.drop(labels="class" , axis = 1)
y = Data["class"]
X.head()
Out[10]:
Performing scaling on dataset using Standardscalar¶
In [11]:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X)
X = scaler.transform(X)
Task2: Understanding the attributes - Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why (10 points)¶
Relationship between the attributes¶
In [36]:
Data.columns
Out[36]:
Covariance of all variables¶
The column 'compactness' and 'pr.axis_aspect_ratio' have small covariance as well as correlation. The column 'compactness' and the columns 'distance_circularity', 'scatter_ratio','elongatedness', 'pr.axis_rectangularity', 'scaled_variance', 'scaled_variance.1'are highly correlated and covariance is also high above 0.75. Also other columns have very small covariance which we can observe in the below code output.
In [38]:
cv = Data.cov()
In [45]:
sorted_cv = cv.sort_values(by='compactness', axis=0, ascending=False)
sorted_cv
Out[45]:
In [12]:
covMatrix = np.cov(X,rowvar=False)
print(covMatrix)
In [ ]:
from the correlation matrix we can observe few columns are highly related to each other.¶
Correlation matrix¶
In [13]:
plt.figure(figsize = (12,12))
def plot_corr(X, size=11):
corr = X.corr()
sns.heatmap(corr, annot=True)
plot_corr(Data)
In [14]:
sns.pairplot(Data,diag_kind='kde')
Out[14]:
In [15]:
columns = list(Data)[0:-1] # Excluding Outcome column
Data[columns].hist(stacked=False, bins=100, figsize=(12,30), layout=(14,2));
In [ ]:
Task3: Split the data into train and test (Suggestion: specify “random state” if you are using train_test_split from Sklearn) (5 marks)¶
In [16]:
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42,stratify = y)
# 42 is just any random seed number
y_train.value_counts()/y_train.shape[0]*100
Out[16]:
In [17]:
y_train.value_counts().plot(kind='bar');
In [18]:
y_test.value_counts()/y_test.shape[0]*100
Out[18]:
In [19]:
y_test.value_counts().plot(kind='bar');
In [ ]:
Task4: Train a Support vector machine using the train set and get the accuracy on the test set (10 marks)¶
In [20]:
svm_classifier = svm.SVC(gamma=0.025, C=3)
In [21]:
svm_classifier.fit(x_train , y_train)
Out[21]:
In [22]:
y_pred = svm_classifier.predict(x_test)
In [23]:
print("Accuracy %0.2f " % (svm_classifier.score(x_train, y_train)))
print("Accuracy %0.2f " % (svm_classifier.score(x_test, y_test)))
In [ ]:
In [ ]:
Task5: Perform K-fold cross validation and get the cross validation score of the model (optional)¶
with k-fold cross validation with s_splits = 10, we can get the accuracy of 0.97 with +/- 0.02 i.e. around 95 %
In [50]:
from sklearn.model_selection import cross_val_score
scores = cross_val_score(svm_classifier, X, y, cv=10, scoring='accuracy')
print(scores)
In [57]:
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
In [59]:
import numpy as np
from sklearn.model_selection import KFold
cv = KFold(n_splits=10, random_state=1, shuffle=True)
In [60]:
cv
Out[60]:
In [61]:
scores = cross_val_score(svm_classifier, X, y, scoring='accuracy', cv=cv, n_jobs=-1)
In [62]:
scores
Out[62]:
In [63]:
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
Task 6: Use PCA from Scikit learn, extract Principal Components that capture about 95% of the variance in the data – (10 points)¶
In [24]:
pca = PCA(n_components=18)
pca.fit(X)
Out[24]:
In [25]:
print(pca.explained_variance_)
In [26]:
print(pca.components_)
In [ ]:
In [27]:
print(pca.explained_variance_ratio_)
In [28]:
plt.bar(list(range(1,19)),pca.explained_variance_ratio_,alpha=0.5, align='center')
plt.ylabel('Variation explained')
plt.xlabel('eigen Value')
plt.show()
In [29]:
plt.step(list(range(1,19)),np.cumsum(pca.explained_variance_ratio_), where='mid')
plt.ylabel('Cum of variation explained')
plt.xlabel('eigen Value')
plt.show()
In [30]:
pca7 = PCA(n_components=7)
pca7.fit(X)
print(pca7.components_)
print(pca7.explained_variance_ratio_)
Xpca7 = pca7.transform(X)
In [31]:
Xpca7
Out[31]:
In [32]:
sns.pairplot(pd.DataFrame(Xpca7))
Out[32]:
Task 7: Repeat steps 3,4 and 5 but this time, use Principal Components instead of the original data. And the accuracy score should be on the same rows of test data that were used earlier. (hint: set the same random state) (20 marks)¶
In [33]:
xpca_train, xpca_test, ypca_train, ypca_test = train_test_split(Xpca7, y, test_size=0.3, random_state=42)
In [34]:
clf_pca = svm.SVC(gamma=0.025, C=1000)
clf_pca.fit(xpca_train , ypca_train)
print("Train Accuracy %0.2f " % (clf_pca.score(xpca_train, ypca_train)))
print("Test Accuracy %0.2f " % (clf_pca.score(xpca_test, ypca_test)))
Task 8: Compare the accuracy scores and cross validation scores of Support vector machines – one trained using raw data and the other using Principal Components, and mention your findings (5 points)¶
In [ ]:
In [35]:
print("Accuracy %0.2f " % (svm_classifier.score(x_train, y_train)))
print("Accuracy %0.2f " % (svm_classifier.score(x_test, y_test)))
print("with PCA Train Accuracy %0.2f " % (clf_pca.score(xpca_train, ypca_train)))
print("with PCA Test Accuracy %0.2f " % (clf_pca.score(xpca_test, ypca_test)))
Observing the results above, the accuracy score with PCA with dimensianality reduction is good (test accuracy 0.94 ) closer to the accuracy without PCA and dimensionality reduction (test accuracy of 0.97).¶
In [ ]: